Many of image pickup apparatuses in these days are being provided with an image pickup element having an extremely large number of pixels from several million to several tens of million pixels for picking up a high resolution image.
However, it is uncommon to utilize a display apparatus that can output a high resolution image corresponding to the number of pixels provided to the image pickup element, and a situation also occurs in which if the high resolution image output from the image pickup element is recorded in a memory as it is, a memory capacity necessary for the recording is increased, and the number of images that can be recorded is decreased.
By taking the above-mentioned circumstances into account, in the image pickup apparatus provided with the image pickup element having the large number of pixels, an output pixel signal from the image pickup element is not recorded in the memory as it is, and a processing of reducing the total number of pixels to be recorded in the memory is often carried out through a processing of thinning out the number of output pixels or a synthesis processing including an addition computation of plural pixels and the like.
A related art technology disclosing the above-mentioned pixel number reduction processing includes, for example, PTL 1 (Japanese Unexamined Patent Application Publication No. 2008-278453).
With reference to FIG. 1, an outline of the processing described in PTL 1 will be described.
FIG. 1 illustrates
(a) a pixel arrangement of an image pickup element, and
(b) a circuit configuration of the image pickup element
which are described in the above-mentioned PTL 1.
It should be noted that the circuit illustrated in FIG. 1(b) is a circuit corresponding to a pixel region 11 for a part of four pixels including Gb1, R1, Gb2, and R2 illustrated in FIG. 1(a).
According to one of embodiments described in PTL 1, in the image pickup element configuration illustrated in FIG. 1(a), the pixel region 11 is set as a common pixel region, that is, a unit region including plural pixels for deciding a pixel value of one pixel constituting an output image as the pixel number reduction image. The pixels of Gb and R are connected via transfer transistors (T1 to T4) to a gate part of a transistor for amplification (AMP) as illustrated in FIG. 1(b).
To obtain the output image where the number of pixels is reduced, by using the circuit configuration illustrated in FIG. 1(b), an addition computation of pixels having a same color included in the region 11 (according to the present example, Gb1 and Gb2) is carried out.
FIG. 2 illustrates a timing chart of a pixel value addition processing executed by using the circuit configuration illustrated in FIG. 1(b).
FIG. 2 illustrates these signal patterns of
transfer transistor signals: T1 to T4,
a reset signal: RST (reset of floating diffusion (FD)), and
a selection signal: SEL, in
(1) a shutter operation for regulating an exposure start, and
(2) a read out operation for regulating an exposure end.
FIG. 2 illustrates a signal pattern in a case where an addition processing for the pixel Gb1 and the pixel Gb2 illustrated in FIG. 1(a) is carried out.
A period between times ts and te illustrated in FIG. 2 is equivalent to an exposure period.
In response to ON of the transistors T1 and T3 in the shutter operation of (1) in FIG. 2, an exposure for the pixel Gb1 and the pixel Gb2 illustrated in FIG. 1(a) is started. After that, in response to ON of the transistors T1 and T3 in the read out operation of (2) in FIG. 2, the exposure for the pixel Gb1 and the pixel Gb2 illustrated in FIG. 1(a) is ended, and the read out processing is executed.
In an initial stage (T=t1) of the read out processing, the selection (SEL) of the common pixel and the reset (RST) of the floating diffusion (FD) are carried out, and subsequently, at a time (T=t2), the transistors T1 and T3 of the pixel Gb1 and the pixel Gb2 illustrated in FIG. 1(a) are read out at the same time, and electrons generated in Gb1 and Gb2 are accumulated in the floating diffusion (FD) for the addition. Thus, an addition signal based on pixel values of the two pixels is obtained. An average signal of the plural pixels is calculated on the basis of these addition signals, for example, and an output pixel value is calculated.
According to PTL 1, while the above-mentioned addition processing for the plural pixel is used as a basic configuration, for example, a pixel number reduction configuration through a pixel value addition while four pixels are set as one pixel as illustrated in FIG. 3 is disclosed. FIG. 3 illustrates these drawings of
(a) a pixel array of the image pickup element, and
(b) a pixel gravity center of the output pixel.
The pixel array illustrated in FIG. 3(a) is a pixel array similar to the pixel array illustrated in FIG. 1(a). This pixel array is equivalent to a pixel array of the image pickup element.
On the basis of four pixels corresponding to pixels having the same color in the image pickup element of FIG. 3(a), a pixel value of one pixel of the output image is set and output.
That is, the total number of pixels is reduced by consolidating the four pixels into one pixel for the output.
The pixel gravity center of output pixel of FIG. 3(b) illustrates a pixel gravity center in the original image pickup element with regard to each of the output pixels after the reduction in the number of pixels.
For example, a Gb pixel 31 of FIG. 3(b) corresponds to a pixel value decided by evenly using pixel values of Gb pixels on four corners in a 3×3 pixel block 21 of FIG. 3(a), and a pixel gravity center is set at a center position of the 3×3 pixel block 21. This gravity center position is represented by the Gb pixel 31 of FIG. 3(b)
An example illustrated in FIG. 3 is an example in which a processing of reducing the number of pixels of the output image into ¼ of the number of pixels of the image pickup element is carried out while 64 pixels of 8×8 pixels illustrated in (a) is set as 4×4=16 pixels illustrated in (b).
For this processing, for example, the addition processing on the (Gb) pixels on the four corners of the 3×3 pixel block 21 illustrated in FIG. 3(a) is executed, and the pixel value of one Gb pixel in the output image is calculated.
The pixel values of the two Gb in a vertical direction are added while following the circuit described above with reference to FIG. 1(b). After that, two addition values output from the image pickup element in every other column are further added to calculate an addition value of the four Gb pixel values, and thereafter, an average value or the like based on these addition values is calculated to decide the pixel value of one pixel of the output image based on the four pixel.
That is, on the basis of the four Gb pixels included in the 3×3 pixel block 21 illustrated in FIG. 3(a), the pixel value of the Gb pixel 31 illustrated in FIG. 3(b) is calculated.
In this case, the gravity center of the Gb pixel 31 in the output image is at a position of (x, y)=(2, 2) in coordinate axes where a horizontal right direction is set as x and a vertical downward direction is set as y, that is, a position of the Gb pixel 31 illustrated in FIG. 3(b).
Also, on the basis of the four B pixels included in a 3×3 pixel block 22 illustrated in FIG. 3(a), a pixel value of a B pixel 32 illustrated in FIG. 3(b) is calculated.
In this case, a gravity center of the B pixel 32 in the output image is at a position of (x, y)=(3, 2), that is, a position of the B pixel 32 illustrated in FIG. 3(b).
Similarly, on the basis of the four Gb pixels included in a 3×3 pixel block 23 illustrated in FIG. 3(a), a pixel value of a GB pixel 33 illustrated in FIG. 3(b) is calculated.
In this case, a gravity center of the Gb pixel 33 in the output image is at a position of (x, y)=(6, 2), that is, a position of the B pixel 33 illustrated in FIG. 3(b).
Also, on the basis of the four B pixels included in a 3×3 pixel block 24 illustrated in FIG. 3(a), a pixel value of a B pixel 34 illustrated in FIG. 3(b) is calculated.
In this case, a gravity center of the B pixel 34 in the output image is at a position of (x, y)=(7, 2), that is, a position of the B pixel 34 illustrated in FIG. 3(b).
The total 16 pixels illustrated in FIG. 3(b) are output as an image of 4×4 pixels in a case where the pixels are set as an output image.
That is, the pixels are output as an image 70 of 4×4 pixels as illustrated in FIG. 4(c).
FIG. 4 illustrates these
(b) a pixel gravity center of the output pixel (same as FIG. 3(b)), and
(c) a pixel position of the output image.
In FIG. 4(c), a consideration is given while 2×2 pixels on the upper left, that is, a block of 2×2 pixels including the Gb pixel 31 and the B pixel 32 is fixed. In the case of this setting, the other three 2×2 pixel blocks are all moved in accordance with arrows (α), (β), and (γ) illustrated in FIG. 4(c) and output as constitutional pixels of the image 70 of 4×4 pixel illustrated in FIG. 4(c).
Through this shift processing, the following problem occurs.
For example,
the Gb pixel 33 where the position of the pixel gravity center is at (x, y)=(6, 2) is set as a Gb pixel 53 where the pixel position is at (x, y)=(3, 2) in the output image.
Also,
the B pixel 34 where the position of the pixel gravity center is at (x, y)=(7, 2) is set as a B pixel 54 where the pixel position is at (x, y)=(3, 3) in the output image.
Here, a reduction scale rate is calculated.
A consideration is given while it is assumed that the GB pixel 31 at the pixel position (x, y)=(2, 2) is set as a reference pixel at a fixed position.
The Gb pixel 33 at the pixel position (x, y)=(6, 2) illustrated in FIG. 4(b) is away by 4 pixels from the Gb pixel 31 corresponding to the reference pixel.
In the output image, since this is set as the Gb pixel 53 where the pixel position is (x, y)=(3, 2), a distance from the reference pixel: the Gb pixel 31 is 2 pixels.
That is, the reduction scale rate is
2 pixels/4 pixels=½.
On the other hand, the B pixel 34 at the pixel position (x, y)=(7, 2) illustrated in FIG. 4(b) is away by 5 pixels from the Gb pixel 31 corresponding to the reference pixel.
In the output image, since this is set as the B pixel 54 where the pixel position is (x, y)=(4, 2), a distance from the reference pixel: the Gb pixel 31 is 3 pixels.
That is, the reduction scale rate is
3 pixels/5 pixels=⅗.
In this manner, the reduction scale rates between the pixels fluctuate, and an output image having a relative position different from relative positions of the respective pixels of the picked-up image corresponding to the pixel array of the image pickup element is generated.
That is, the output image is generated while spacings between the respective pixels of an original image picked up in the image pickup element are unevenly reduced.
The above-mentioned unevenness in the pixel spacings causes an image quality degradation.
To be specific, for example, a degradation such as an expansion of jaggies illustrated in FIG. 5 occurs.
The original image of FIG. 5(A) is a high resolution image with a large number of pixels equivalent to the picked-up image of the image pickup element, and this image has small jaggies.
If the pixel number reduction processing with which the relative pixel positions described with reference to FIG. 1 to FIG. 4 are set to be varied is carried out on the basis of this original image of FIG. 5(A), an image in which the jaggies are expanded as illustrated in FIG. 5 (B) is generated.
It should be noted that the jaggies are a type of folding noise. Because of the unevenness in the spacings between the pixel gravity centers after the addition, the jaggy degradation is increased.
It should be noted that in addition to this, a disarray of positional relationships between respective color arrangements of RGB or the like causes various image quality degradations in which a difference also occurs in colors between the output image and the original image.